The test manager determines how the PRA will be performed. This is done in parallel to or after the previous step. Two choices must be made when determining the PRA approach:

• How will the PRA be organised, in sessions or interviews?
• Which risk classification is to be used?

These aspects are discussed in greater detail below.

Organisation of the PRA

The PRA can be performed in the following ways:

1. The test manager preferably organises one session – but more if needed – with all of the participants
2. The test manager takes individual interviews with the participants instead of a session.
3. There are various intermediate possibilities. The test manager may, for instance, organise a session for each group of participants (e.g. users and IT people). Another variant is to do interviews each with 2 or 3 participants instead of a session. The participants must complement each other in this case (e.g. a user and a developer).

A session can also be held with the main participant group, and additional interviews taken with participants with specific expertise. Since a PRA can require a lot of manpower in a more complex environment, another intermediate form is to do a first session with a limited number of participants (preferably with good knowledge of the product to be discussed), after which a more complete team supplements, corrects and approves the results in a follow-up session.

A session has a maximum duration of half a day, an interview has a maximum duration of 2 hours. 

Session or interview

Generally sessions are preferred over interviews, in particular due to the shared commitment they inspire. But there are also good reasons to take interviews. Below you will find a number of aspects that have an impact on the choice:

• Need for shared commitment - A session is greatly preferred over interviews to highlight the various perspectives of the participants and work towards a shared and widely supported vision.
• (Un)familiarity with PRA and risk-based testing - If some participants are not familiar with PRA and testing, this might disturb a session because a lot of explanation is required. A prior kick-off session to explain the PRA is necessary in this case. If this is impossible, it is better to use interviews.
• Number of participants - If there is a large group of participants (> 8), the risk that timid persons will or cannot express their view increases in a session. Doing multiple sessions or interviews is recommended in this case.
• Political tension - There are two choices when such tension exists: either the test manager and client want to make the tension visible and discussible; or they want to move around it. In the first case the session is the best option.
• Group thinking versus individual thinking - A session inspires cross-fertilisation of ideas and thoughts, with the whole being more than the sum of the parts. On the other hand, some individuals are heard insufficiently in sessions. The test manager must make a decision on the basis of the participants to be involved.
• Little overhead versus thorough approach - Sessions give an impression of a thorough approach with a lot of (too much) overhead, in view of the (large) number of participants stuck together for several hours. When the organisation has a clear preference for “lean and mean”, this would be an argument to take interviews.
• “Discussion culture” - While a session always involves the risk of getting mired in discussions on minor details, the risk is higher in some organisations than in others. In these cases, interviews are the preferred option.
• Personal preferences and competencies - Does the test manager like to take interviews, or does he prefer moderating a session? An interview is easier than a session. How much experience does he have with the latter?

If the test manager opts for a session, he must select a session technique. Techniques like metaplan or the similar but somewhat less formal brown paper sessions are extremely suitable.

Metaplan

Metaplan techniques are tools to gather ideas with a group of people in a short time. The method was initiated by Eberhard Schnelle in Hamburg. In addition to simple visual techniques, like the use of boards with cards stuck on them, the method uses moderators to facilitate the discussion and a structured preparation process (with adequate question definition being vital) through to conclusion and assessment of the results. Many years of experience with metaplan sessions are required to become an experienced moderator. Less experienced moderators (and test managers) can implement a simplified version of the technique successfully. It is characteristic of the method to collect ideas from different perspectives, focusing more on categorising and prioritising the main ideas than striving towards completeness. In a PRA, a test manager can use the metaplan technique to determine the test goals together, inventory the characteristic/object part combinations to be tested, and classify the associated risks.

A checklist for an interview agenda can be found on http://www.tmap.net/.

Before and during the session or interview, inspiration can also be found in checklists, for instance “Checklist risk factors per quality characteristic” (see http://http//www.tmap.net) and experiences from previous tests. FMEA (Failure Mode and Effect Analysis) is a separate, formal approach in which the participants analyse, for all functions of the product, what could go wrong and what the consequences would be. Additional (test) measures are defined for major risks. See www.fmeainfocentre.com for more information.

The result of the process, which is described in more detail in 4. Collecting And Analysing Product Risks, is recorded.

Alternative: Combining PRA with test strategy

When the number of participants involved in the PRA is limited and they already have experience with PRA, or when the product risks are fairly easy to classify, it may be efficient to combine the PRA in one session with the test strategy (see sections Determine The Test Strategy (MTP) and Determine The Test Strategy (AST)).

Kick-off session for maintenance testing

When inventorying the test basis, it often happens that changes are not properly documented. In the case of ad-hoc maintenance, the concrete cause/reason may not be specified. One proven method to clarify this is to organise a kick-off session with all relevant parties (functional and technical administrators, developers, users, testers). In this session, the PRA is preceded by an impact determination and followed by the creation or adjustment of the test strategy. Attention is also devoted to non-functional quality characteristics. Take the existing situation into account when gathering nonfunctional quality characteristics. For instance, toughening a performance requirement from three seconds to one second is generally difficult to realise by means of maintenance if this was not taken into account in the original design of the system.

Specifically in the case of ad-hoc maintenance, it can be discussed in a kick-off session how a defect in a test situation can be reproduced. The challenge in organising a kick-off session is aligning the agendas of the required participants with the available – usually limited – test lead time.

Determining risk classification method

To determine what the more or less risky characteristics and components of the product to be tested are, a classification method is necessary to specify whether something represents a high or a low risk. The various risks can all be valued separately (absolute classifi cation), or they can be valued in relation to each other (relative classification). These two risk classification methods are discussed in greater detail below. In most cases, we recommend that an organisation should choose one risk classification mechanism. However, when the risks of the systems are highly diversified, e.g. in administrative and safetycritical systems, it is better to make the classification application-dependent.

Absolute classification

In this classification method, it is determined for each separate risk how big the damage in case of failure and the chance of failure are. The scale of the damage and the chance are plotted against each other, resulting in the risk class. An example is the table below [Broekman and Notenboom, 2003], where A stands for a high risk, B for medium, and C for low risk.

The risk classes (A, B and C) in the table above are not distributed symmetrically. Application in actual practice has shown that many organisations feel it is more important to control a risk with high damage and low chance of failure than a risk with low damage and high chance of failure. Clearly an organisation can create the table at its own discretion.

For a small test object or in an organisation that has little experience with risk analyses, a system distinguishing only between the categories High and Low is usually adequate. The associated risk classes are shown in the table below [Lyndsay, 2002]:

So-called detail risk factors can be used as a possible intermediary step to determine the damage in case of failure and the chance of failure. This means that the damage in case of failure and the chance of failure are elaborated in greater detail. It is important here to determine in advance how the scores for the detail risk factors will determine the risk class. A few examples to clarify this:

Example 1 - The stakeholders state that for system X, the non-functioning of function Y represents a risk when the system is taken live.

Detail risk factors in relation to the damage in case of failure for function Y are:

• The output is clearly visible to the client (damage H)
• The chance of loss of image is high (damage H)
• The impact on other functions is low (damage L).

Detail risk factors in relation to the chance of failure for function Y are:

• The function is used very often (chance of failure H)
• The function is simple (chance of failure L)
• The developers are very experienced (chance of failure L).

The stakeholders have reached the following agreement:

• If a H is scored for 2 detail factors, the rating is High
• If a H is scored for 1 detail factor or an M for two detail factors, the rating is Medium
• Else the rating is Low.

For the example above, this means that the Damage in case of failure is assessed as H (High). The Chance of failure in this example is M (Medium). These values result in a risk class B.

Example 2 - Another organization has based the classification on Damage categories:

High: >250,000 euros
Medium: from 50,000 to 250,000 euros
Low: < 50,000 euros

Relative classification

In this classification method, the various product risks are placed in their ‘order of importance’ in relation to each other. The observed risks are analysed with a number of stakeholders. It is then determined where the risk must be placed in relation to the others. This results in a list of risks, with the importance of the risk determining the order. An example to clarify this:

Example 3 - The stakeholders recognize the following risks for system X:

1. The performance of the night batch is too slow, meaning that employees do not have access to the system in the morning.
2. The invoices that are sent to the customers contain incorrect amounts.
3. The list of check totals for the internal accounting department contains incorrect data.

The stakeholders analyze the various risks in a meeting and agree on the order 2, 1, 3. They feel that risk 2 is the most important one.

This classification method is commonly used in practice for small test objects or in organisations with little experience in performing risk analyses.

Tips - In a situation in which many risks are discerned (e.g. 20), we recommend not using an order from 1 through 20. The danger is that the order of the risks will be subject to too much discussion. In such a case it is better to work with a number of groups. For instance three groups (high, medium and low importance) in which the risks are placed.